Methods and apparatus for environmental measurements and/or stereoscopic image capture

ABSTRACT

A camera rig including one or more stereoscopic camera pairs and/or one or more light field cameras are described. Images are captured by the light field cameras and stereoscopic camera pairs are captured at the same time. The light field images are used to generate an environmental depth map which accurately reflects the environment in which the stereoscopic images are captured at the time of image capture. In addition to providing depth information, images captured by the light field camera or cameras is combined with or used in place of stereoscopic image data to allow viewing and/or display of portions of a scene not captured by a stereoscopic camera pair.

RELATED APPLICATIONS

U.S. Provisional Patent Application Ser. No. 62/106,122 filed Jan. 21,2015, U.S. Provisional Patent Application Ser. No. 62/257,651 filed Nov.19, 2015 and U.S. Provisional Patent Application Ser. No. 62/260,238filed Nov. 25, 2015 are hereby expressly incorporated by reference intheir entirety.

FIELD

The present invention relates to methods and apparatus for makingenvironmental measurements and/or capturing stereoscopic images, e.g.,pairs of left and right eye images.

BACKGROUND

Stereoscopic playback devices, e.g., display devices which are capableof displaying different images to a users left and right eyes, aregrowing in popularity. Unfortunately, there is limited stereoscopiccontent available for such devices at the present time. This is due, inpart to the difficulty with existing camera systems and rigs to captureimages that are well suited for presentation as stereoscopic images witha realistic 3D effect.

Rather than capture two images in parallel, e.g., a left and right eyeimage, many systems have take the approach of capturing images usingmultiple cameras each oriented in a different direction and then usingcomputations to simulate 3D effects and generate left and right eyeimages for playback. In such cases left and right eye images, e.g.,stereoscopic image pairs, are not captured in parallel but rathergenerated through relatively complicated computational processingintended to generate image pairs.

While the computational approach to generating stereoscopic imagecontent from images captured by cameras facing different directions canhave some advantages in that the camera rig need not simulate the humanvisual system and/or in terms of the number of cameras needed thecomputational processing associated with such an approach has certaindisadvantages in terms of the quality of the 3D content which isgenerated and is also not well suited for real time content capture andstreaming to playback devices given the number and time involved withgenerating pairs of left and right eye images from the content capturedby cameras which are arranged in a configuration very different from thespacing and/or orientation of a normal human's pair of eyes.

In addition to capturing stereoscopic image content, depth measurementsare also desirable so that an accurate model of an environment can begenerated and used during playback. While static environmental modelsmay be used it would be highly desirable if environmental measurements,e.g., depth measurements relative to a camera position, could be madeduring an event and used to generate or update an environmental module.

It would be desirable if environmental measurements could be made duringan event from the same rig used to capture stereoscopic camera images sothat the measurement accurately reflect distances, e.g., depths.

In view of the above, it should be appreciated that there is a need forimproved methods and/or apparatus for capturing and/or processingstereoscopic image content. In addition there is a need for methodsand/or apparatus for capturing environmental information, e.g., depthinformation, which can be used for generating or updating a 3D model ofan environment. It should be appreciated that to be beneficial ordesirable a device need not support both stereoscopic image capture andenvironmental measurements but it would be desirable if in at least someembodiments a camera rig could capture stereoscopic image content aswell as environmental measurement information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a camera rig implemented in accordance with oneembodiment along with a calibration target which may be used to forcalibrating the camera rig.

FIG. 2 illustrates the camera rig with three pairs of cameras, e.g., 3pairs of cameras capturing stereoscopic image data, mounted in thecamera rig.

FIG. 3 illustrates an exemplary camera rig with an exemplary protectivecover implemented in accordance with some exemplary embodiments.

FIG. 4 illustrates another exemplary camera rig implemented inaccordance with an exemplary embodiment with various elements of thecamera rig being shown for clarity in partially disassembled form.

FIG. 5 shows the camera rig of FIG. 4 with the cameras mounted thereonalong with an audio capture device including ear shaped devicesincluding microphones used for capturing stereo audio.

FIGS. 6-9 illustrate various views of an exemplary camera rigimplemented in accordance with some exemplary embodiments.

FIG. 10 illustrates a front view of an exemplary arrangement of an arrayof cameras that can be used in the exemplary camera rigs of the presentinvention such as camera rigs shown in FIGS. 1-9, in accordance withsome embodiments.

FIG. 11 illustrates a front view of yet another exemplary arrangement ofan array of cameras that can be used in any of the camera rigs of thepresent invention.

FIG. 12A is a first part of FIG. 12 which illustrates a flowchart of anexemplary method of operating an imaging apparatus in accordance withsome embodiments.

FIG. 12B is a second part of FIG. 12 which illustrates a flowchart of anexemplary method of operating an imaging apparatus in accordance withsome embodiments.

FIG. 12, shows how FIGS. 12A and 12B in combination comprise FIG. 12.

FIG. 13 illustrates an exemplary light field camera which can be used inthe camera rigs shown in FIGS. 1-9.

FIG. 14 illustrates an exemplary processing system implemented inaccordance with the invention.

FIG. 15A illustrates a perspective view of an exemplary tower mountedsingle camera pair rig suitable for capturing images intended forstereoscopic viewing.

FIG. 15B illustrates a front view of the exemplary tower mounted singlestereo camera pair rig.

FIG. 15C is a drawing illustrating a side view of the exemplary towermounted single stereo camera pair rig of FIG. 15A.

FIG. 15D is a drawing illustrating a top view of the exemplary singletower stereo camera pair rig of FIG. 15A.

FIG. 16A illustrates a top view of an exemplary triple stereo camerapair rig in accordance with some embodiments.

FIG. 16B is a drawing illustrating a perspective view of the exemplarytriple stereo camera pair rig with various features and/or elements ofthe triple stereo camera pair rig being shown in more detail.

FIG. 16C is a drawing illustrating a side view of the exemplary triplestereo camera pair rig.

FIG. 17 illustrates an exemplary tri stereo camera rig in accordancewith an exemplary embodiment.

FIG. 18 includes two different views of an exemplary tri stereo camerarig that includes a top upward facing camera and a bottom downwardfacing camera in accordance with an exemplary embodiment.

FIG. 19 illustrates an exemplary two level stereo camera rig thatincludes six pairs of stereoscopic cameras arranged three camera pairsper level.

FIG. 20A illustrates a view of an exemplary quad stereoscopic camerapair rig in accordance with an exemplary embodiment.

FIG. 20B is a drawing illustrating a top view of the exemplary quadstereoscopic camera pair rig of FIG. 20A with more constructionsfeatures and/or dimensions shown for further detail.

FIG. 21A illustrates a view of an exemplary penta (five) stereoscopiccamera pair rig in accordance with an exemplary embodiment.

FIG. 21B is a drawing illustrating a top view of the exemplary pentastereoscopic camera pair rig with more constructions features and/ordimensions shown for further detail.

FIG. 22 is a drawing illustrating a side by side comparison of theexemplary penta stereoscopic camera pair rig on the left and theexemplary quad stereo camera pair rig shown on the right.

FIG. 23 illustrates a view of an exemplary hexa (six) stereoscopiccamera pair rig in accordance with an exemplary embodiment.

FIG. 24 is a drawing illustrating a top view of the hexa (six)stereoscopic camera pair rig with more constructions features and/ordimensions shown for further detail.

FIG. 25 illustrates a top view of an exemplary bi-level ninestereoscopic camera pair rig in accordance with an exemplary embodiment.

FIG. 26 illustrates an exemplary support structure, e.g., tripod andsupport ring or plate, which can be used to support various exemplarycamera rigs including, for example, the three sided camera rig shown inFIG. 3.

FIG. 27 illustrates an exemplary four legged support structure which canbe used to support one or more of the exemplary camera rigs shown inother figures including, for example, the four sided camera rigs shownin FIGS. 8 and 9.

FIG. 28 shows an exemplary stereoscopic camera rig including 3 pairs ofcameras and a preferred orientation of the camera rig to the supportlegs of the tripod support structure shown in FIG. 26 that is used insome but not all embodiments.

FIG. 29 shows an exemplary camera rig including 4 pairs of cameras usedfor stereoscopic image capture and a preferred orientation of the camerarig to the support legs of the four legged support structure shown inFIG. 27 that is used in some but not all embodiments.

FIG. 30 illustrates how the camera rig and support structure shown inFIG. 29 may appear during use with the camera rig secured to the supportstructure.

SUMMARY

Methods and apparatus for capturing stereoscopic image content and/ormaking environmental measurements are described.

In at least some embodiments a camera rig including one or more pairs ofcameras are used. In one embodiment a camera rig includes at least afirst camera pair including first and second cameras used to captureleft and right eye images in parallel. The cameras in the first camerapair in some embodiments are arranged in parallel facing in a firstdirection and are operated in parallel. The spacing between the camerapairs in some but not necessarily all embodiments is intended toapproximate the spacing between a human's pair of eyes. In someembodiments multiple pairs of cameras are arranged on a camera rig withthe different pairs of cameras being spaced in a horizontal plane tocapture a 360 degree view of the environment.

The spacing between camera pairs maybe and in some embodiments isuniform. In some embodiment not only is the spacing between camera pairsthe same, the spacing between cameras is also intentionally arranged sothe distance between the optical axis of one camera to the optical axisof another camera in the horizontal plane is the same whether thedistance between optical axis is being measured between adjacent camerasof a camera pair or between adjacent cameras of different camera pairs.Such uniform spacing is not required for all embodiments but is used inat least some embodiments.

In some but not all embodiments an upward camera or camera pair maybeinclude d in addition to the multiple horizontal outward facing camerapairs. In addition in some embodiments a downward camera or camera pairmaybe included in addition to the vertical camera or camera pair and/orhorizontal camera pairs. To allow capture of images of the ground thelens or lens assembly of the downward facing camera may extend through asupport ring used to secure the camera rig to a support structure suchas a tripod or four legged stand.

In some embodiments the legs of the camera support structure are alignedwith the interface between camera pairs with lenses facing in agenerally horizontal outward facing direction. In this manner, the legswill appear in peripheral portions of images captured by cameras whichare facing in a generally horizontal direction and not block or appearat the center of images captured by such cameras. Legs of the supportstructure maybe of a predetermined color. The use of a predeterminedcolor for the legs facilitates removal of pixels corresponding to thelegs from captured images. In some embodiments portions of the legswhich are captured by a camera are removed or concealed by processingthe captured images before streaming of the captured image content to aplayback device or devices. The portions of the leg maybe replaced withpixel values from adjacent non-leg portions of an image being processedand/or concealed through other techniques such as blurring.

By using multiple pairs of cameras to capture left and right eye images,mounted on a camera single camera rig, it is possible to capturestereoscopic image content, e.g., pairs of left and right eye images ina synchronized manner in multiple directions, e.g., directions thatmaybe need to generate a complete or nearly complete 360 degree worldview from the position of the camera rig.

Because image pairs are captured, less processing maybe required than insystems where stereoscopic image pairs are synthesized via computationalapproaches from camera views captured by cameras oriented in different,e.g., non-parallel, directions. Thus, the camera rig of the presentinvention is well suited for capturing image content intended to bestreamed in real or near real time, e.g., live content of a sportingevent, concert or another event. Furthermore, because multiple camerapairs are mounted on a single camera rig, the rig assembly is relativelyeasy to set up and transport. Furthermore, the rig assembly is wellsuited for mounting on a tripod or other movable base making forrelatively easy deployment at field locations including sporting events,concerts, etc. Given that the rig provides for fixed camera spacingrelationships and a fixed height from the ground, computationalprocessing and combining of images can be performed in a manner thattakes into consideration the known rig configuration and camera spacingarrangement.

While various embodiments are well suited for capturing stereo imagepairs in multiple directions in a synchronized manner in real time, thecamera rig in some embodiments is also well equipped for makingenvironmental measurements, e.g., depth measures from the camera rigthrough the use of a light field camera. In some embodiments, inaddition to one or more stereoscopic camera pairs, a light field cameraor array of light field cameras is oriented in the same direction as oneor more cameras used to capture left and right eye images. The lightfield cameras are used to measure depth from the camera rig to objectsin the field of view of the light field camera. The depths measures areused in some embodiments to update an environmental model while an eventis ongoing. In such embodiments the light field cameras provide depthinformation and the environmental module can be updated in response tochanges in the position of objects. The updated environmental depth mapinformation generated from one or more light field camera iscommunicated to a playback device in some cases and used to update anenvironmental map upon which captured images are displayed as textures.Thus, in some embodiments the camera rig can capture depth informationwhich can be used to update an environmental module during an event.

It should be appreciated that with its ability to capture pairs of leftand right eye images in multiple directions in a synchronized manner andto update an environmental map using depth information obtained fromimages captured by one or more light field cameras included on thecamera rig in some embodiments, the camera rig of the present inventionis well suited for supporting real time content capture and streamingfor virtual reality playback devices.

Numerous benefits and features are discussed in the detailed descriptionwhich follows.

DETAILED DESCRIPTION

The present invention is related to the field of panoramic stereoscopicimagery and more particularly, to an apparatus suitable for capturinghigh-definition, high dynamic range, high frame rate stereoscopic,360-degree panoramic video using a minimal number of cameras in anapparatus of small size and at reasonable cost while satisfying weight,and power requirements for a wide range of applications.

Stereoscopic, 360-degree panoramic video content is increasingly indemand for use in virtual reality displays. In order to producestereoscopic, 360-degree panoramic video content with 4K or greater ofresolution, which is important for final image clarity, high dynamicrange, which is important for recording low-light content, and highframe rates, which are important for recording detail in fast movingcontent (such as sports), an array of professional grade, large-sensor,cinematic cameras or of other cameras of suitable quality is oftenneeded.

Camera methods and apparatus including camera apparatus and/or methodswhich are well suited for capturing stereoscopic image data, e.g., pairsof left and right eye images are described. Various features relate tothe field of panoramic stereoscopic imagery and more particularly, to anapparatus suitable for capturing images, e.g., high-definition videoimages. The images may have a high dynamic range, high frame rate, andin some embodiments support 360-degree panoramic video. A camera rigimplemented in accordance with various features may use one or morepairs of cameras and/or a camera pair in combination with one or moresingle cameras. The rig allows for a minimal number of cameras to beused for a given application in an apparatus of small size and atreasonable cost while satisfying weight, and power requirements for awide range of applications. In some embodiments a combination ofstereoscopic cameras and Light Field cameras (also referred to as Lytrocameras) arranged in a specific manner is used.

Stereoscopic, 360-degree panoramic video content is increasingly indemand for use in 3D stereoscopic playback systems and/or virtualreality displays. In order to produce stereoscopic, 360-degree panoramicvideo content, e.g., with 4K, or greater of resolution, which isimportant for final image clarity, high dynamic range, which isimportant for recording low-light content, and high frame rates, whichare important for recording detail in fast moving content (such assports), an array of professional grade, large-sensor, cinematic camerasor of other cameras of suitable quality is often needed.

In order for the camera array to be useful for capturing 360-degree,stereoscopic content for viewing in a stereoscopic virtual realitydisplay, the camera array should acquire the content such that theresults approximate what the viewer would have seen if his head wereco-located with the camera. Specifically, the pairs of stereoscopiccameras should be configured such that their inter-axial separation iswithin an acceptable delta from the accepted human-model average of 63mm (millimeters). Additionally, the distance from the panoramic array'scenter point to the entrance pupil of a camera lens (aka nodal offset)should be configured such that it is within an acceptable delta from theaccepted human-model average of 101 mm (millimeters).

In order for the camera array to be used to capture events and spectatorsports where it should be compact and non-obtrusive, it should beconstructed with a relatively small physical footprint allowing it to bedeployed in a wide variety of locations and shipped in a reasonablesized container when shipping is required. The camera array should alsobe designed, if possible, such that the minimum imaging distance of thearray is small, e.g., as small as possible, which minimizes the “deadzone” where scene elements are not captured because they fall outside ofthe field of view of adjacent cameras. The camera rig of the presentinvention show in FIG. 1 and various other embodiments addresses one ormore of these design goals.

It would be advantageous if the camera array included in the rig can becalibrated for optical alignment by positioning calibration targetswhere the highest optical distortion is prone to occur (where lensangles of view intersect and the maximum distortion of the lensesoccur). To facilitate the most efficacious calibration targetpositioning, target locations should, and in some embodiments are,determined formulaically from the rig design.

FIG. 1 shows an exemplary camera configuration used in some embodiments.The support structure shown in FIGS. 4 and 5 is not shown in FIG. 1 toallow for better appreciation of the camera pair arrangement shown usedin some embodiments.

While in some embodiments three camera pairs are used such as in theFIG. 1 example in some but not all embodiments a camera array, e.g., thecamera positions of the rig, is populated with only 2 of the 6-totalcameras which maybe used to support simultaneous 360-degree stereoscopicvideo. When the camera rig or assembly is configured with less than all6 cameras which can be mounted in the rig, the rig is still capable ofcapturing the high-value, foreground 180-degree scene elements inreal-time while manually capturing static images of the lower-value,background 180-degree scene elements, e.g., by rotating the rig when theforeground images are not being captured. For example, in someembodiments when a 2-camera array is used to capture a football gamewith the field of play at the 0-degree position relative to the cameras,the array is manually rotated around the nodal point into the 120-degreeand 240-degree positions. This allows the action on the field of asports game or match, e.g., foreground, to be captured in real time andthe sidelines and bleachers, e.g., background areas, to be captured asstereoscopic static images to be used to generate a hybridized panoramaincluding real time stereo video for the front portion and static imagesfor the left and right rear portions. In this manner, the rig can beused to capture a 360 degree view with some portions of the 360 viewbeing captured at different points in time with the camera rig beingrotated around its nodal axis, e.g., vertical center point between thedifferent points in time when the different view of the 360 scene areaare captured. Alternatively, single cameras may be mounted in the secondand third camera pair mounting positions and mono (non-stereoscopic)video captured for those areas.

In other cases where camera cost is not an issue, more than two camerascan be mounted at each position in the rig with the rig holding up to 6cameras as in the FIG. 1 example. In this manner, cost effect cameradeployment can be achieved depending on the performance to be capturedand, the need or ability of the user to transport a large number, e.g.,6 cameras, or the user's ability to transport fewer than 6 cameras,e.g., 2 cameras.

FIG. 1 depicts a six (6) camera assembly 100 also sometimes referred toas a rig or camera array, along with a calibration target 115. Thecamera rig 100 illustrated in FIG. 1 includes a support structure (shownin FIGS. 4 and 5) which holds the cameras in the indicated positions, 3pairs 102, 104, 106 of stereoscopic cameras (101, 103), (105, 107),(109, 111) for a total of 6 cameras. The support structure includes abase 720 also referred to herein as a mounting plate (see element 720shown in FIG. 4) which supports the cameras and to which plates on whichthe cameras are mounted can be secured. The support structure maybe madeof plastic, metal or a composite material such as graphite orfiberglass, and is represented by the lines forming the triangle whichis also used to show the spacing and relationship between the cameras.The center point at which the doted lines intersect represents thecenter nodal point around which the camera pairs 102, 104, 106 can berotated in some but not necessarily all embodiments. The center nodalpoint corresponds in some embodiments to a steel rod or threaded centermount, e.g., of a tripod base, around which a camera support framerepresented by the triangular lines can be rotated. The support framemay be a plastic housing in which the cameras are mounted or tripodstructure as shown in FIGS. 4 and 5.

In FIG. 1, each pair of cameras 102, 104, 106 corresponds to a differentcamera pair position. The first camera pair 102 corresponds to a 0degree forward to front facing position and normally meant to cover theforeground where the main action occurs. This position normallycorresponds to the main area of interest, e.g., a field upon which asports game is being played, a stage, or some other area where the mainaction/performance is likely to occur. The second camera pair 104corresponds to a 120 degree camera position (approximately 120 degreefrom the front facing) degree position) and is used to capture a rightrear viewing area. The third camera pair 106 corresponds to a 240 degreeviewing position (approximately 240 degree from the front facing) and aleft rear viewing area. Note that the three camera positions are 120degrees apart.

Each camera viewing position includes one camera pair in the FIG. 1embodiment, with each camera pair including a left camera and a rightcamera which are used to capture images. The left camera captures whatare sometimes referred to as a left eye images and the right cameracaptures what is sometime referred to as right eye images. The imagesmay be part of a view sequence or still image captured at one or moretimes. Normally at least the front camera position corresponding tocamera pair 102 will be populated with high quality video cameras. Theother camera positions may be populated with high quality video cameras,lower quality video cameras or a single camera used to capture still ormono images. In some embodiments the second and third camera embodimentsare left unpopulated and the support plate on which the cameras aremounted is rotated allowing the first camera pair 102 to capture imagescorresponding to all three camera positions but at different times. Insome such embodiments left and right rear images are captured and storedand then video of the forward camera position is captured during anevent. The captured images may be encoded and streamed in real time,e.g. while an event is still ongoing, to one or more playback devices.

The first camera pair 102 shown in FIG. 1 includes a left camera 101 anda right camera 103. The left camera has a first lens assembly 120secured to the first camera and the right camera 103 has a second lensassembly secured to the right camera 103. The lens assemblies 120, 120′include lenses which allow for a wide angle field of view to becaptured. In some embodiments each lens assembly 120, 120′ includes afish eye lens. Thus each of the cameras 102, 103 can capture a 180degree field of view or approximately 180 degrees. In some embodimentsless than 180 degrees is captured but there is still at least someoverlap in the images captured from adjacent camera pairs in someembodiments. In the FIG. 1 embodiment a camera pair is located at eachof the first (0 degree), second (120 degree), and third (240 degree)camera mounting positions with each pair capturing at least 120 degreesor more of the environment but in many cases with each camera paircapturing 180 degrees or approximately 180 degrees of the environment.

Second and third camera pairs 104, 106 are the same or similar to thefirst camera pair 102 but located at 120 and 240 degree camera mountingpositions with respect to the front 0 degree position. The second camerapair 104 includes a left camera 105 and left lens assembly 122 and aright camera 107 and right camera lens assembly 122′. The third camerapair 106 includes a left camera 109 and left lens assembly 124 and aright camera 111 and right camera lens assembly 124′.

In FIG. 1, D represents the inter-axial distance of the first 102stereoscopic pair of cameras 101, 103. In the FIG. 1 example D is 117 mmwhich is the same or similar to the distance between pupils of the leftand right eyes of an average human being. Dashed line 150 in FIG. 1depicts the distance from the panoramic array's center point to theentrance pupil of the right camera lens 120′ (aka nodal offset). In oneembodiment corresponding to the FIG. 1 which example the distanceindicated by reference number 150 is 315 mm but other distances arepossible.

In one particular embodiment the footprint of the camera rig 100 isrelatively small. Such a small size allows the camera rig to be placedin an audience, e.g., at a seating position where a fan or attendancemight normally be located or positioned. Thus in some embodiments thecamera rig is placed in an audience area allowing a viewer to have asense of being a member of the audience where such an effect is desired.The footprint in some embodiments corresponds to the size of the base towhich the support structure including, in some embodiments a centersupport rod is mounted or support tower is located. As should beappreciated the camera rigs in some embodiments can rotate around thecenter point of the base which corresponds to the center point betweenthe 3 pairs of cameras. In other embodiments the cameras are fixed anddo not rotate around the center of the camera array.

The camera rig is capable of capturing relatively close as well asdistinct object. In one particular embodiment the minimum imagingdistance of the camera array is 649 mm but other distances are possibleand this distance is in no way critical.

The distance from the center of the camera assembly to the intersectionpoint 151 of the views of the first and third camera parts represents anexemplary calibration distance which can be used for calibrating imagescaptured by the first and second camera pairs. In one particularexemplary embodiment, an optimal calibration distance, where lens anglesof view intersect and the maximum distortion of the lenses occur is 743mm. Note that target 115 may be placed at a known distance from thecamera pairs located at or slightly beyond the area of maximumdistortion. The calibration target include a known fixed calibrationpattern. The calibration target can be and is used for calibrating thesize of images captured by cameras of the camera pairs. Such calibrationis possible since the size and position of the calibration target isknown relative to the cameras capturing the image of the calibrationtarget 115.

FIG. 2 is a diagram 200 of the camera array 100 shown in FIG. 1 ingreater detail. While the camera rig 100 is again shown with 6 cameras,in some embodiment the camera rig 100 is populated with only twocameras, e.g., camera pair 102 including cameras 101 and 103. As shownthere is a 120 degree separation between each of the camera pairmounting positions. Consider for example if the center between eachcamera pair corresponds to the direction of the camera mountingposition. In such a case the first camera mounting position correspondsto 0 degrees, the second camera mounting position corresponds to 120degrees and the third camera mounting position corresponding to 240degrees. Thus each camera mounting position is separated by 120 degrees.This can be seen if the center line extending out through the center ofeach camera pair 102, 104, 106 was extended and the angle between thelines measured.

In the FIG. 2 example, the pair 102, 104, 106 of cameras can, and insome embodiments do, rotate around the center point of the camera rigallowing for different views to be captured at different times withouthaving to alter the position of the camera rig base. That is, thecameras can be rotated around the center support of the rig and allowedto capture different scenes at different times allowing for a 360 degreescene capture using the rig shown in FIG. 2 while it is populated withonly two cameras. Such a configuration is particularly desirable from acost perspective given the cost of stereoscopic cameras and is wellsuited for many applications where it may be desirable to show abackground captured from the same point of view but at a different timethan the time at which the front scene including the main action duringa sporting event or other event may occur. Consider for example thatduring the event objects may be placed behind the camera that it wouldbe preferable not to show during the main event. In such a scenario therear images may be, and sometimes are, captured prior to the main eventand made available along with the real time captured images of the mainevent to provide a 360 degree set of image data.

FIG. 3 shows an exemplary camera rig 300 which is the same or similar tothe rig of FIGS. 1 and 2 but without a support tripod and with a plasticcover 350 placed over the camera pairs. The plastic cover 350 includeshandles 310, 312, 314 which can be used to lift or rotate, e.g., whenplaced on a tripod, the camera rig 300. The camera rig 300 is shown withthree pairs of cameras, a first camera pair 302 including cameras 301,303 with lens assemblies 320, 320′, a second camera pair 304 includingcameras with lens assemblies 322, 322′, and a third camera pair 306including cameras with lens assemblies 324, 324′. The plastic cover 350is secured to the mounting platform 316, which may be implemented as aflat plate with one or more slots and screw holes as shown in FIG. 4.The plastic cover 350 is secured to the base with nuts or screws 330,331 which can be removed or tightened by hand to allow for easy removalor attachment of the cover 350 and easy access to the cameras of thecamera pairs. While six cameras are included in the rig 300 shown inFIG. 3, a single camera pair may be included and/or a single camera pairwith one or more individual cameras located at the other camera mountingpositions where the camera pairs are not mounted may be used.

FIG. 4 is a detailed diagram of a camera rig assembly 400 shown inpartially disassembled form to allow better view of how the componentsare assembled. The camera rig 400 is implemented in accordance with oneexemplary embodiment and may have the camera configuration shown inFIGS. 1 and 2. In the example shown in FIG. 4 various elements of thecamera rig 400 are shown in disassembled form for clarity and detail. Ascan be appreciated from FIG. 4, the camera rig 400 includes 3 pairs ofcameras 702, 704 and 706, e.g., stereoscopic cameras, which can bemounted on a support structure 720 of the camera rig 400. The first pairof cameras 702 includes cameras 750 and 750′. The second pair of cameras704 includes cameras 752. 752′ and the third pair of cameras 706includes cameras 754, 754′. The lenses 701, 701′ of the cameras 750,750′ can be seen in FIG. 7. While elements 701 and 701′ are described aslenses, in some embodiments they are lens assemblies which are securedto the cameras 750, 750 with each lens assembly including multiplelenses positioned in a lens barrel which is secured to the cameras 750,750′ via a friction fit or twist lock connection.

In some embodiments the three pairs (six cameras) of cameras 702, 704and 706 are mounted on the support structure 720 via the respectivecamera pair mounting plates 710, 712 and 714. The support structure 720may be in the form of a slotted mounting plate 720. Slot 738 isexemplary of some of the slots in the plate 720. The slots reduce weightbut also allow for adjustment of the position of the camera mountingplates 710, 712, 714 used to support camera pairs or in some cases asingle camera.

The support structure 720 includes three different mounting positionsfor mounting the stereoscopic camera pairs 702, 704, 706, with eachmounting position corresponding to a different direction offset 120degrees from the direction of the adjacent mounting position. In theillustrated embodiment of FIG. 7, the first pair of stereoscopic cameras702 is mounted in a first one of the three mounting positions, e.g.,front facing position, and corresponds to a front viewing area. Thesecond pair 704 of stereoscopic cameras 704 is mounted in a second oneof the three mounting positions, e.g., background right positionrotating 120 degrees clockwise with respect the front position, andcorresponds to a different right rear viewing area. The third pair 706of stereoscopic cameras is mounted in a third one of the three mountingpositions, e.g., background left position rotating 240 degrees clockwisewith respect the front position, and corresponds to a left rear viewingarea. The cameras in each camera position capture at least a 120 viewingarea but may capture in many case at least a 180 degree viewing arearesulting in overlap in the captured images which can facilitiescombining of the images into a 360 degree view with some of theoverlapping portions being cut off in some embodiments.

The first camera pair mounting plate 710 includes threaded screw holes741, 741′, 741″ and 741′″ through which screws 704, 740′, 740″, 740″ canbe inserted, respectively through slots 738 and 738; to secure the plate710 to the support structure 720. The slots allow for adjustment of theposition of the support plate 710.

The cameras 750, 750′ of the first camera pair are secured to individualcorresponding camera mounting plates 703, 703′ using screws that passthrough the bottom of the plates 703, 703′ and extend into threadedholes on the bottom of the cameras 750, 750′.

Once secured to the individual mounting plates 703, 703′ the cameras750, 750′ and mounting plates 703, 703′ can be secured to the camerapair mounting plate 710 using screws. Screws 725, 725′, 725″ (which isnot fully visible) and 725′″ pass through corresponding slots 724 intothreaded holes 745, 745′, 745″ and 745′″ of the camera pair mountingplate 710 to secure the camera plate 703 and camera 750 to the camerapair mounting plate 710. Similarly, screws 727, 727′(which is not fullyvisible), 727″ and 727″ pass through corresponding slots 726, 726′, 726″and 726′″ into threaded holes 746, 746′, 746″ and 746′″ of the camerapair mounting plate 710 to secure the camera plate 703′ and camera 750′to the camera pair mounting plate 710.

The support structure 720 has standoff rollers 732, 732′ mounted toreduce the risk that an object moving past the support structure willget caught on the support structure as it moves nearby. This reduces therisk of damage to the support structure 720. Furthermore by having ahollow area inside behind the roller an impact to the roller is lesslikely to be transferred to the main portion of the support structure.That is, the void behind the rollers 732, 732′ allows for somedeformation of the bar portion of the support structure on which thestandoff roller 732′ is mounted without damage to the main portion ofthe support structure including the slots used to secure the cameramounting plates.

In various embodiments the camera rig 400 includes a base 722 to whichthe support structure 720 is rotatable mounted e.g. by a shaft orthreaded rod extending trough the center of the base into the supportplate 720. Thus in various embodiments the camera assembly on thesupport structure 720 can be rotated 360 degrees around an axis thatpasses through the center of the base 722. In some embodiments the base722 may be part of a tripod or another mounting device. The tripodincludes legs formed by pairs of tubes (742, 742′), (742″ and 742″) aswell as additional leg which is not visible in FIG. 4 due to the viewingangle. The legs are secured by a hinge to the base 722 and can be foldedfor transport. The support structure maybe made of plastic, metal or acomposite material such as graphite or fiberglass or some combinationthereof. The camera pairs can be rotated around a central point,sometimes referred to as center nodal point, in some embodiments.

The assembly 400 shown in FIG. 4 allows for the position of individualcameras to be adjusted from the top by loosing the screws securing theindividual camera mounting plates to the camera pair mounting plate andthen adjusting the camera position before retightening the screws. Theposition of a camera pair can be adjusted by moving the camera pairmounting plate after loosening the screws accessible from the bottomside of the support structure 720, moving the plate and thenretightening the screws. Accordingly, what the general position anddirection of the camera pairs is defined by the slots in the supportplate 720, the position and direction can be finely adjusted as part ofthe camera calibration process to achieve the desired camera alignmentwhile the cameras are secured to the support structure 720 in the fieldwhere the camera rig is to be used.

In FIG. 5 reference numbers which are the same as those used in FIG. 4refer to the same elements. FIG. 5 illustrates a drawing 500 showing theexemplary camera rig 400 in assembled form with additional stabilizationplates 502, 502′, 504, 504′, 506 and stabilization plate joining bars503, 505, 507, 509, 511, 513 added to the tops of the camera pairs toincrease the rigidity and stability of the cameras pairs after they havebeen adjusted to the desired positions.

In the drawing 500 the camera pairs 702, 704, 706 can be seen mounted onthe support structure 720 with at least one of the camera pair mountingplate 710 being visible in the illustrated drawing. In addition to theelements of camera rig 400 already discussed above with regard to FIG.4, in drawing 500 two simulated ears 730, 732 mounted on the camera rigcan also be seen. These simulated ears 730, 732 imitate human ears andin some embodiments are made from silicone or plastic molded in theshape of a human ear. Simulated ears 730, 732 include microphones withthe two ears being separated from each other by a distance equal to, orapproximately equal to, the separation between human ears of an averagehuman. The microphones mounted in the simulated ears 730, 732 aremounted on the front facing camera pair 702 but could alternatively bemounted on the support structure, e.g., platform, 720. The simulatedears 730, 732 are positioned perpendicular to the front surface of thecamera pair 702 in a similar manner as human ears are positionedperpendicular to the front surface of eyes on a human head. Holes in theside of the simulated ears 730, 732 act as an audio/sound entry point tothe simulated ears with the simulated ears and hole operating incombination to direct audio towards a microphone mounted in each one ofthe simulated ears much as a human ear directs audio sounds into theeardrum included in a human ear. The microphones in the left and rightsimulated ears 730, 732 provide for stereo sound capture similar to whata human at the location of the camera rig 500 would perceive via thehuman's left and right ears if located at the position of the camerarig. The audio input of the microphones mounted in the simulate ears isperpendicular to the face of the outer lens of front facing cameras 750,750′ in the same manner that the sensor portion of a human ear would besomewhat perpendicular to the humans beings face. The simulate earsdirect sound into toward the microphone just as a human ear would directsound waves towards a human ear drum.

The simulated ears 730, 730 are mounted on a support bar 510 whichincludes the microphones for capturing sound. The audio capture system730, 732, 810 is supported by a movable arm 514 which can be moved viahandle 515.

While FIGS. 4-5 illustrate one configuration of an exemplary camera rigwith three stereoscopic camera pairs, it should be appreciated thatother variations are possible. For example, in one implementation thecamera rig 400 includes a single pair of stereoscopic cameras which canrotate around the center point of the camera rig allowing for different120 degree views to be captured at different times. Thus a single camerapair can be mounted on the support structure and rotated around thecenter support of the rig and allowed to capture different scenes atdifferent times allowing for a 360 degree scene capture.

In other embodiments the camera rig 400 includes a single stereoscopiccamera pair 702 and one camera mounted in each of the second and thirdpositions normally used for a pair of stereoscopic cameras. In such anembodiment a single camera is mounted to the rig in place of the secondcamera pair 704 and another single camera is mounted to the camera rigin place of the camera pair 706. Thus, in such an embodiment, the secondcamera pair 704 may be thought of as being representative of a singlecamera and the camera pair 706 may be thought of as being illustrativeof the additional single camera.

FIGS. 6-9 illustrate various views of other exemplary camera rigsimplemented in accordance with some exemplary embodiments.

FIG. 6 illustrates a drawing 800 showing one view of an exemplary camerarig 801 implemented in accordance with some exemplary embodiments. Anarray of cameras is included in the camera rig 801 some of which arestereoscopic cameras. In the illustrated view of the camera rig 801 indrawing 800, only a portion of the camera rig 801 is visible while asimilar arrangement of cameras exist on the other sides (also referredto as different faces) of the camera rig 801 which cannot be fully seenin the drawing 800. In some but not all embodiments, the camera rig 801includes 13 cameras secured by a top plastic body or cover 805 and abottom base cover 842. In some embodiments 8 of these 13 cameras arestereoscopic cameras such as the cameras 804, 806, 812 and 814 in pairswhile many other cameras are light field cameras such as cameras 802 and810 which are visible in the drawing 800 and cameras 815 and 820 whichare not fully but partially visible in drawing 800. Various othercombinations of the cameras are possible. In some embodiments a camera825 is also mounted on the top portion of the camera rig 801, e.g., topface 840 of camera rig 801, to capture images of a top hemisphere of anenvironment of interest. The plastic body/cover 805 includes handles811, 813, 817 which can be used to lift or rotate the camera rig 801.

In some embodiments the camera rig 801 includes one light field camera(e.g., camera 802) and two other cameras (e.g., cameras 804, 806)forming a stereoscopic camera pair on each longer side of the camera rig801. In some such embodiments there are four such longer sides (alsoreferred to as the four side faces 830, 832, 834 and 836) with eachlonger side having one light field camera and one stereoscopic camerapair, e.g., light field camera 802 and stereoscopic camera pair 804, 806on one longer side 836 to the left while another light field camera 810and stereoscopic camera pair 812, 814 on the other longer side 830 tothe right can be seen in drawing 800. While the other two side faces arenot fully shown in drawing 800, they are shown in more detail in FIG. 8.In some embodiments at least some of the cameras, e.g., stereoscopiccameras and the light field cameras, in the camera rig 801 use a fisheye lens. In various embodiments each of the cameras in the camera rig801 is protected by a corresponding lens/camera guard to protect thecamera and/or lens against a physical impact and/or damage that may becaused by an object. For example cameras 802, 804 and 806 are protectedby guards 845, 847 and 849 respectively. Similarly cameras 810, 812 and814 are protected by guards 850, 852 and 854 respectively.

In addition to the stereoscopic camera pair and the light field cameraon each of the four side faces 830, 832, 834 and 836, in someembodiments the camera rig 801 further includes a camera 825 facing inthe upward vertical direction, e.g., towards the sky or another topceiling surface in the case of a closed environment, on the top face 840of the camera rig 801. In some such embodiments the camera 825 on thetop face of the camera rig 801 is a light field camera. While not shownin drawing 800, in some other embodiments the top face 840 of the camerarig 801 also includes, in addition to the camera 825, anotherstereoscopic camera pair for capturing left and right eye images. Whilein normal circumstances the top hemisphere (also referred to as the skyportion) of a 360 degree environment, e.g., stadium, theater, concerthall etc., captured by the camera 825 may not include action and/orremain static in some cases it may be important or desirable to capturethe sky portion at the same rate as other environmental portions arebeing captured by other cameras on the rig 801.

While one exemplary camera array arrangement is shown and discussedabove with regard to camera rig 801, in some other implementationsinstead of just a single light field camera (e.g., such as cameras 802and 810) arranged on top of a pair of stereoscopic cameras (e.g.,cameras 804, 806 and 812, 814) on four faces 830, 832, 834, 836 of thecamera rig 801, the camera rig 801 includes an array of light fieldcameras arranged with stereoscopic camera pair. For example in someembodiments there are 3 light field cameras arranged on top of astereoscopic camera pair on each of the longer sides of the camera rig801. In another embodiment there are 6 light field cameras arranged ontop of stereoscopic camera pair on each of the longer sides of thecamera rig 801, e.g., with two rows of 3 light field cameras arranged ontop of the stereoscopic camera pair. Some of such variations arediscussed with regard to FIGS. 12-13. Moreover in another variation acamera rig of the type shown in drawing 800 may also be implemented suchthat instead of four faces 830, 832, 834, 836 with the cameras pointedin the horizontal direction as shown in FIG. 8, there are 3 faces of thecamera rig with cameras pointing in the horizontal direction.

In some embodiments the camera rig 801 may be mounted on a supportstructure such that it can be rotated around a vertical axis. In variousembodiments the camera rig 801 may be deployed in an environment ofinterest, e.g., such as a stadium, auditorium, or another place where anevent to be captured is taking place. In some embodiments the lightfield cameras of the camera rig 801 are used to capture images of theenvironment of interest, e.g., a 360 degree scene area of interest, andgenerate depth maps which can be used in simulating a 3D environment anddisplaying stereoscopic imaging content.

FIG. 7 illustrates a drawing 900 showing the exemplary camera rig 801with some elements of the camera rig 801 being shown in a disassembledform for more clarity and detail. Various additional elements of thecamera rig 801 which were not visible in the illustration shown indrawing 800 are shown in FIG. 7. In FIG. 7, same reference numbers havebeen used to identify the elements of the camera rig 801 which wereshown and identified in FIG. 6. In drawing 900 at least the two sidefaces 830 and 836 as well as the top face 840 and bottom face 842 of thecamera rig 801 are visible.

In drawing 900 various components of the cameras on two out of four sidefaces 830, 832, 834, 836 of the camera rig 801 are shown. The lensassemblies 902, 904 and 906 correspond to cameras 802, 804 and 806respectively of side face 836 of the camera rig 801. Lens assemblies910, 912 and 914 correspond to cameras 810, 812 and 814 respectively ofside face 830 while lens assembly 925 corresponds to camera 825 on thetop face of the camera rig 801. Also show in drawing 900 are three sidesupport plates 808, 808′, and 808′″ which are support the top and bottomcover plates 805 and 842 of the camera rig 801. The side support plates808, 808′, and 808′″ are secured to the top cover 805 and bottom basecover 842 via the corresponding pairs of screws shown in the Figure. Forexample the side support plate 808 is secured to the top and bottomcover plates 805, 842 via the screw pairs 951 and 956, the side supportplate 808′ is secured to the top and bottom cover plates 805, 842 viathe screw pairs 952 and 954, and the side support plate 808′″ is securedto the top and bottom cover plates 805, 842 via the screw pairs 950 and958. The camera rig 801 in some embodiments includes a base support 960secured to the bottom cover plate 842 via a plurality of screws 960. Insome embodiments via the base support 960 the camera rig may be mountedon a support structure such that it can be rotated around a verticalaxis, e.g., axis going through the center of base 960. The externalsupport structure may be a tripod or another platform.

FIG. 8 illustrates a drawing 1000 showing a top view of the exemplarycamera rig 801 with more elements of the camera rig 801 being shown ingreater detail. In the top view of the camera rig 801 the other two sidefaces 832 and 834 which were not fully visible in drawings 800-900 aremore clearly shown. The lens assemblies 915, 916 and 918 correspond tocamera 815 and the stereoscopic camera pair on the side face 832 of thecamera rig 801. Lens assemblies 920, 922 and 924 correspond to camera920 and the stereoscopic camera pair on the side face 834 of the camerarig 801.

As can be seen in drawing 1000, the assembly of cameras on each of thefour sides faces 830, 832, 834, 836 (small arrows pointing towards thefaces) and the top face 840 of the camera rig 801 face in differentdirections. The cameras on the side faces 830, 832, 834, 836 of thecamera rig 801 are pointed in the horizontal (e.g., perpendicular to thecorresponding face) while the camera(s) on the top face 840 is pointedin the upward vertical direction. For example as shown in FIG. 8 thecameras on the face 836 of the camera rig 801 (cameras corresponding tolens assemblies 902, 904, 906) are facing in a first direction shown byarrow 1002. The arrow 1004 shows a second direction in which the camerason the face 830 of the camera rig 801 (cameras corresponding to lensassemblies 910, 912, 914) are facing, arrow 1006 shows a third directionin which the cameras on the face 832 of the camera rig 801 (camerascorresponding to lens assemblies 915, 916, 918) are facing, arrow 1008shows a fourth direction in which the cameras on the face 834 of thecamera rig 801 (cameras corresponding to lens assemblies 920, 922, 924)are facing and arrow 1010 shows a fifth (vertical) direction in whichthe camera on the top face 840 of the camera rig 801 (camera 825corresponding to lens assembly 925, is facing. In various embodimentsthe first, second, third and fourth directions are generally horizontaldirections while the fifth direction is a vertical direction. In someembodiments the cameras on the different side faces 830, 832, 834 and836 are uniformly spaced. In some embodiments the angle between thefirst, second, third and fourth directions is the same. In someembodiments the first, second, third and fourth directions are differentand 90 degrees apart. In some other embodiments the camera rig isimplemented such that instead of four side faces the camera rig has 3side faces with the same or similar camera assemblies as shown indrawings 800-1000. In such embodiments the cameras on the side faces ofthe camera rig 801 point in three different directions, e.g., a first,second and third direction, with the first, second and third directionsbeing 120 degrees apart.

FIG. 11 illustrates a drawing 1100 showing a view of yet anotherexemplary camera rig 1101 implemented in accordance with some exemplaryembodiments. The exemplary camera rig 1101 is similar to the camera rig801 in most and many aspects and includes the same or similarconfiguration of cameras as discussed with regard to camera rig 801above. The camera rig 1101 includes four side faces 1130, 1132, 1134,1136 and a top face 1140 similar to camera rig 801. Each of the fourside faces 1130, 1132, 1134, 1136 of the camera rig 1101 includes anarray of cameras including a light field camera and a pair ofstereoscopic camera pair while the top face 1140 of camera rig includesat least one camera device 1125 similar to what has been shown anddiscussed with regard to camera rig 801. However the camera rig 1101further includes, in addition to the camera arrays on each of the fivefaces 1130, 1132, 1134, 1136 and 1140, a sixth bottom face 1142including at least one camera 1126 facing vertically downward, e.g.,towards the ground. In some such embodiments the bottom surface camera1126 facing vertically downwards and the top face camera 1125 facingvertically upwards are light field cameras. In some embodiments each ofthe cameras 1125 and 1126 are part of a corresponding stereoscopiccamera pair on the top and bottom faces 1140, 1142 of the camera rig1101.

While the stereoscopic cameras of the camera rigs 801 and 1101 are usedto capture stereoscopic imaging content, e.g., during an event, the useof light field cameras allows for scanning the scene area of interestand generate depth maps of various portions of the scene area capturedby the light field cameras (e.g., from the captured images correspondingto these portions of the scene of interest). In some embodiments thedepth maps of various portions of the scene area may be combined togenerate a composite depth map of the scene area. Such depth maps and/orcomposite depth map may, and in some embodiments are, provided to aplayback device for use in displaying stereoscopic imaging content andsimulating a 3D environment which can be experienced by the viewers.

The use of light field camera in combination with the stereoscopiccameras allows for environmental measurements and generation of theenvironmental depth maps in real time, e.g., during an event being shot,thus obviating the need for deployment of equipment for environmentalmeasurements to be performed offline ahead in time prior to the start ofan event, e.g., a football game and/or other performance in anenvironment.

While the depth map generated from each image captured by the lightfield camera corresponds to a portion of the environment to be mapped,in some embodiments the depth maps generated from individual images areprocessed, e.g., stitched together, to form a composite map of thecomplete environment scanned using the light field cameras. Thus byusing the light field cameras a relatively complete environmental mapcan be, and in some embodiments is generated.

In the case of light field cameras, an array of micro-lenses capturesenough information that one can refocus images after acquisition. It isalso possible to shift, after image capture, one's viewpoint within thesub-apertures of the main lens, effectively obtaining multiple views. Inthe case of a light field camera, depth cues from both defocus andcorrespondence are available simultaneously in a single capture. Thiscan be useful when attempting to fill in occluded information/sceneportions not captured by the stereoscopic cameras.

The depth maps generated from the light field camera outputs will becurrent and is likely to accurately measure changes in a stadium orother environment of interest for a particular event, e.g., a concert orgame to be captured by a stereoscopic camera. In addition, by measuringthe environment from the same location or near the location at which thestereoscopic camera are mounted, the environmental map, at least in someembodiments, accurately reflects the environment as it is likely to beperceived from the perspective of the stereoscopic cameras that are usedto capture the event.

In some embodiments images captured by the light field cameras can beprocessed and used to fill in for portions of the environment which arenot captured by a stereoscopic camera pair, e.g., because the positionand/or field of view of the stereoscopic camera pair may be slightlydifferent from that of the light field camera and/or due to anobstruction of view from the stereoscopic cameras. For example, when thelight field camera is facing rearward relative to the position of thestereoscopic pair it may capture a rear facing view not visible to aforward facing stereoscopic camera pair. In some embodiments output ofthe light field camera is provided to a playback device separately oralong with image data captured by the stereoscopic camera pairs. Theplayback device can use all or portions of the images captured by thelight field camera when a scene area not sufficiently captured by thestereoscopic camera pairs is to be displayed. In addition a portion ofan image captured by the light field camera may be used to fill in aportion of the a stereoscopic image that was occluded from view from theposition of the stereoscopic camera pair but which a user expects to beable to see when he or she shifts, e.g., slightly rotate and/or tilt,his or her head to the left or right relative to the default viewingposition corresponding to the location of the stereoscopic camera pair.For example, if a user leans to the left or right in an attempt to peekaround a column obstructing his/her view, in some embodiments contentfrom one or more images captured by the light field camera will be usedto provide the image content which was not visible to or captured by thestereoscopic camera pair but which is expected to be visible to the userfrom the shifted head portion the user achieves during playback byleaning left or right.

FIG. 12 illustrates a front view of an exemplary arrangement 1200 of anarray of cameras that can be used in an exemplary camera rig implementedin accordance with the invention such as camera rig 300, camera rig 400and/or camera rigs 801 and 1101 in accordance with some embodiments. Incomparison to the arrangement shown in drawing 800 with a single lightfield camera arranged on top of a pair of stereoscopic cameras on eachof the faces of the camera rig 801, the exemplary arrangement 1200 usesan array of light field cameras 1202, 1204 and 1206 arranged with astereoscopic camera pair 1208, 1210. The exemplary arrangement 1200 maybe, and in some embodiments is, used in a camera rig (such as camera rig801) implemented in accordance with the invention. In such embodimentseach face of the camera rig uses the exemplary arrangement 1200 withthree light field cameras (e.g., 1202, 1204 and 1206) arranged with asingle pair of stereoscopic cameras (e.g., 1208, 1210). It should beappreciated that many variations in arrangement are possible and arewithin the scope of the invention.

FIG. 13 illustrates a front view of yet another exemplary arrangement1300 of an array of cameras that can be used in an exemplary camera rigsuch as camera rig 801 or any of the other camera rigs discussedearlier, in accordance with some embodiments. In comparison to thearrangement shown in drawing 800 with a single light field cameraarranged on top of a pair of stereoscopic cameras, the exemplaryarrangement 1300 uses an array of six light field cameras 1302, 1304,1306, 1308, 1310 and 1312 arranged with a stereoscopic camera pair 1320,1322. The light field cameras are stacked in two rows of 3 light fieldcameras arranged one on top of the other with each row including a groupof three light field cameras as shown. The exemplary arrangement 1300may be, and in some embodiments is, used in a camera rig (such as camerarig 801) implemented in accordance with the invention with each face ofthe camera rig using the arrangement 1300.

FIG. 12, which comprises a combination of FIGS. 12A and 12B, illustratesa flowchart 1400 of an exemplary method of operating an imagingapparatus, e.g., the camera rig 1101 shown in FIG. 30 as well as otherfigures and/or another one of the camera rigs in the presentapplication, in accordance with some embodiments. As should beappreciated, in some embodiments the imaging apparatus includes multiplepairs of stereoscopic cameras as well as one or more light fieldcameras. In addition to having camera pairs used to capture left andright eye images for stereoscopic purposes, the camera rig may also haveone or more light field cameras facing outward in the same direction asa corresponding stereo camera pair. An upward and/or downward facingcamera or camera pair may also be included. The upward and downwardfacing cameras may include a light filed camera, a non-light fieldcamera or a combination of light field and non-light field cameras.

As should be appreciated light filed cameras are well suited forgenerating depth maps from the images they capture using know depthestimation techniques.

The method 1400 shown in FIG. 12 which comprise the combination of FIGS.12A and 12B will now be described in detail. The method may beimplemented using the camera rig 1101 which is operated under control ofthe control routines 1614 of the processing system 1600 and whichprovides captured images and/or depth maps produced by light fieldcameras to the processing system for further processing and possibleencoding and streaming of captured image content. The depths maps maybegenerated by the light field cameras and/or by the processing system1600 from images captured by the light field cameras.

The method 1400 begins in step 1402 with the processing system 1600 andcamera rig, e.g., rig 1101, being powered on. Operation proceeds fromstart step 1402 to various image capture steps 1404, 1406 and 1408 whichmaybe, and sometimes are, implemented in parallel synchronized manner sothat images are captured by different stereoscopic camera pairs inaddition to one or more light field cameras in parallel. While imagecapture is synchronized in some embodiments this is not required andlight field image capture and depth map generation will often occur at adifferent frame rate than the rate at which images are captured by thecameras of the stereoscopic camera pairs which normally operate at avideo frame rate, e.g., 30, 60 or some other number of frames persecond.

In step 1404, light field cameras are operated to capture images ofportions of an environment of interest, e.g., all or a portion of anenvironment also being captured by one or more cameras of the stereopairs, upward facing camera or downward facing camera. In step 1410 oneor more images are captured using a first light field camera facing in afirst direction. In step 1412, one or more images are captured by asecod light filed camera facing in a second direction and in step 1414one or more images are captured by a third light field camera facing ina third direction. The first, second and third directions may correspondto different directions, e.g., of the different directions correspondingto a direction in which a stereoscopic camera pair faces for example.The first, second and third light field cameras maybe and sometimes areon different sides of the camera rig 1101. Operation proceeds from step1404 to step 1426 in which the images captured by the light fieldcameras are stored in memory, e.g., memory in the individual light fieldcameras and/or processing system 1600. Then in step 1432 depth maps aregenerated from the images captured by the light field cameras. Eachdepth map corresponds to a portion of the environment of interest. Inthe case where individual cameras store and process the images theycapture, each light field camera would generate a depth map for theportion of the environment in its field of view. In the case where theimage processing system generates the depth maps from captured images,the image processing system 1600 would generate the depth maps fordifferent portions of the environment from the light field images itreceives. If the light field cameras generate the depth maps they aresupplied to the processing system 1600 in sep 1434.

In step 1406 which maybe performed in parallel with step 1404, one ormore stereoscopic camera pairs are operated to capture images, e.g.,left and right eye images one per camera of the camera pair. Step 1406includes step 1416 in which left and right eye images are captured usinga first stereoscopic camera pair. Each camera of the stereoscopic camerapair may and sometimes does include a fish eye lens. Step 1406 may andsometimes does also include step 1418 in which left and right eye imagesare captured using a second stereoscopic camera pair which includesthird and fourth cameras. Step 1406 may and also does include step 1420in some embodiments in which left and right eye images are capturedsuing a third stereoscopic pair including a fifth camera and a sixthcamera. While step 1406 only shows three substeps, in embodiments wherethe camera rig includes more than 3 stereoscopic pairs step 1406 wouldinclude, in some but not necessarily all embodiments, a step ofcapturing images using each of the available stereoscopic pairs. Forexample if the method 1400 is implemented using the camera rig 1101,four stereoscopic camera pairs would be operated in step 1406 to captureimages. In embodiments with a larger number of stereo camera pairs, suchas 5 or 6 camera pairs, step 1406 would include steps for operating 5 or6 stereo camera pairs to capture images. The images captured by thestereo camera pairs are stored in step 1428, e.g., in camera buffermemory and then communicated in step 1434 to the processing system 1600where they are stored and subject to further processing, e.g., encoding,prior to streaming to a playback device. Each of the differentstereoscopic camera pairs may, and in some embodiments does, correspondto a different face of the camera rig on which they are mounted, e.g.,rig 1101. While the rig may include one light field camera per face, insome embodiments the rig includes multiple light field cameras, e.g., anarray of light field cameras per face. The light field cameras maybe inaddition to the stereo camera pair on a face of the rig. Left and rightcameras of a stereoscopic camera pair may, and in various embodimentsdo, have optical axis which extend parallel to each other with thelenses of the camera rig for at least some pairs facing outward.

Step 1408 which may be and sometimes is performed in parallel with steps1404 and 1406. In step 1408 images are captured using one or moreadditional cameras. The cameras may include, for example, an upwardfacing camera and/or a downward facing camera. Images in the upwardand/or downward directions need not be captured in all embodiments butare captured in some embodiments. In the FIG. 12A example, step 1408includes step 1422 in which one or more images are captured by a nithcamera facing in an upward direction and step 1424 in which one or moreimages are captured using a tenth camera facing in a downward direction.Depending on the embodiment the upward facing camera maybe a camera ofan upward facing stereo camera pair, an upward facing light field cameraor a single upward facing non-light field camera. The downward facingcamera maybe a camera of a downward facing stereo camera pair, adownward facing light field camera or a single downward facing non-lightfield camera.

Images captured in step 1408 are stored in step 1430, e.g., in a camerabuffer. In optional step 1433, which is performed in some embodimentswhen the upward and/or downward facing cameras are light field cameras,depth maps are generated by the processors included in the cameras fromthe images captured in step 1408.

In step 1434 the captured images from the upward and/or downward facingcameras along with any generated depth maps are communicated to theprocessing system, e.g., processing system 1600 for generating an updatedepth map representing a 3D environmental model and/or for generatingstereoscopic image content to be streamed, e.g., pairs of left and righteye images corresponding to one or more portions of the environmentcaptured by cameras mounted on the camera rig, e.g., camera rig 1101.

Operation proceeds from step 1434 to step 1438 via connecting node A1436. FIG. 12 includes 1438 as a separate step in which one or moreimages captured by the stereoscopic camera pairs and light filed camerasare communicated to the image processing system 1600. This step may beand sometimes is performed as par to step 1434. Operation proceeds fromstep 1438 to step 1440.

In step 1440 the processing system 1660 is operated to combine depthmaps generated from two or more images captured by different cameras,e.g., light field cameras and/or cameras of the stereoscopic camerapairs, to generate a composite depth map of the environment of interest.The composite depth map is a model of the environment in which theimages where captured since it defines a surface of the environment asviewed from the perspective of the camera rig in the environment.Operation proceeds from step 1440 to step 1442 in which the compositedepth map, e.g., environmental model, of the environment of interest istransmitted to a playback device. The playback device in someembodiments uses the depth map in the rendering of images with capturedand/or transmitted images being applied to the surface of the model aspart of a rendering operation. Thus, from the perspective of a viewerobserving the simulated environment from the position of the camera rig,the viewer will see the images as if they were being observed from thesame size and distance from which they were captured given the user arealistic impression of being in the same environment as the camera rigused to capture the images.

Operation then proceeds to step 1444. In step 1444 the processing system1600 encodes one or more image pairs captured by the stereoscopic camerapairs and transmits it to the playback device. The images maybetransmitted as part of a content stream or streams which are received,decoded and displayed to a user of a playback device taking intoconsideration the environmental model information communicated to theplayback device to be used in image rendering. Thus, a user of aplayback device may receive and view 3D content corresponding to a liveevent while the event is ongoing which might not be possible if a morecomputationally complex method of generating the 3D content stream wasused.

While a 3D content stream is desirable, it maybe desirable to allow auser to view portions of the environment which are occluded from view orcapture by the stereoscopic camera pairs. In step 1450 the processingsystem 1600 transmits at leas a portion of an image generated from aimage captured by a light field camera to the playback device. The imagemaybe an image of the sky or ground not captured by a stereo camera pairor a portion of the environment visible to a light filed camera that wasnot visible to the stereo pair given the position or orientation of thestereo pair. The playback device may, and in some embodiments does, usethe light filed image in a rendering operation to fill in an area of theenvironment which was not captured, e.g., was occluded from view, of astereo camera. The operation shown in FIG. 12 may occur on an ongoingbasis with stereo images and depth map information being captured andupdated on an ongoing basis but potentially at different rates do todifferences between light field and other camera image capture rates.

The ongoing operation of the method shown in FIG. 12 is represented byconnecting node B 1452 showing operation returning form step 1450 tosteps 1404, 1406, 1408.

FIG. 13 illustrates an exemplary light field camera 1500 implemented inaccordance with one exemplary embodiment of the present invention whichcan be used in any of the camera assemblies and/or camera rigs shown inFIGS. 1-11. The camera 1500 can be used to capture images in accordancewith the methods of the present invention and implement one or moresteps of the method of flowchart 1400. The exemplary camera device 1500includes a display device 602, an input device 604, an I/O interface606, a processor 608, memory 610, and a bus 609 which are mounted in ahousing represented by the rectangular box touched by the line leadingto reference number 1500. The camera device 1500 further includes anoptical chain 612 and a network interface 614. The various componentsare coupled together via bus 609 which allows for signals andinformation to be communicated between the components of the camera1500.

The display device 602 may be, and in some embodiments is, a touchscreen, used to display images, video, information regarding theconfiguration of the camera device, and/or status of data processingbeing performed on the camera device. In the case where the displaydevice 602 is a touch screen, the display device 602 serves as anadditional input device and/or as an alternative to the separate inputdevice, e.g., buttons, 606. The input device 604 may be, and in someembodiments is, e.g., keypad, touch screen, or similar device that maybe used for inputting information, data and/or instructions.

Via the I/O interface 606 the camera device 1500 may be coupled toexternal devices and exchange information and signaling with suchexternal devices. In some embodiments via the I/O interface 606 thecamera 1500 may, and in some embodiments does, interfaces with theprocessing system 1600. In some such embodiments the processing system1600 can be used to configure and/or control the camera 1500.

The network interface 614 allows the camera device 1500 to be able toreceive and/or communicate information to an external device over acommunications network.

The optical chain 610 includes a micro lens array 624 and an imagesensor 626. The camera 1500 uses the micro lens array 624 to capturelight information of a scene of interest coming from more than onedirection when an image capture operation is performed by the camera1500.

The memory 612 includes various modules and routines, which whenexecuted by the processor 608 control the operation of the camera 1500in accordance with the invention. The memory 612 includes controlroutines 620 and data/information 622. The processor 506, e.g., a CPU,executes control routines and uses data/information 622 to control thecamera 1500 to operate in accordance with the invention and implementone or more steps of the method of flowchart 400. The processor 608includes a on-chip depth map generation circuit 607 which generatesdepth map of various portions of the environment of interest fromcaptured images corresponding to these portions of the environment ofinterest which are captured during the operation of the camera 1500 inaccordance with the invention. The depth maps of various portions of theenvironment of interest generated by the camera 1500 are stored in thememory 612 as depth maps 630 while images corresponding to one or moreportions of the environment of interest are stored as captured image(s).The captured images and depth maps are stored in memory 612 for futureuse, e.g., additional processing, and/or transmission to another. Invarious embodiments the depth maps 630 and one or more captured images628 are provided to the processing system 104 for further processing andactions in accordance with the features of the invention.

FIG. 14 illustrates an exemplary processing system 1600 in accordancewith the features of the invention. The processing system 1600 can beused to implement one or more steps of the method of flowchart 1400. Theprocessing system 1600 includes multi-rate encoding capability that canbe used to encode and stream stereoscopic imaging content.

The processing system 1600 may be, and in some embodiments is, used toperform composite depth map generation operation, multi-rate encodingoperation, storage, and transmission and/or content output in accordancewith the features of the invention. The processing system 1600 may alsoinclude the ability to decode and display processed and/or encoded imagedata, e.g., to an operator.

The system 1600 includes a display 1602, input device 1604, input/output(I/O) interface 1606, a processor 1608, network interface 1610 and amemory 1612. The various components of the system 1600 are coupledtogether via bus 1609 which allows for data to be communicated betweenthe components of the system 1600.

The memory 1612 includes various routines and modules which whenexecuted by the processor 1608 control the system 1600 to implement thecomposite depth map generation, encoding, storage, andstreaming/transmission and/or output operations in accordance with theinvention. The routines may and sometimes do control image capture byone or more camera rigs such as the multi-camera rig 1101.

The display device 1602 may be, and in some embodiments is, a touchscreen, used to display images, video, information regarding theconfiguration of the processing system 1600, and/or indicate status ofthe processing being performed on the processing device. In the casewhere the display device 602 is a touch screen, the display device 602serves as an additional input device and/or as an alternative to theseparate input device, e.g., buttons, 1606. The input device 1604 maybe, and in some embodiments is, e.g., keypad, touch screen, or similardevice that may be used for inputting information, data and/orinstructions.

Via the I/O interface 606 the processing system 1600 may be coupled toexternal devices and exchange information and signaling with suchexternal devices, e.g., such as the camera rig 801 and/or camera 1500.In some embodiments via the I/O interface 1606 the processing system1600 receives images and depth maps generated by the camera device 1500.

The network interface 1610 allows the processing system 1600 to be ableto receive and/or communicate information to an external device over acommunications network, e.g., such as communications network 105. Thenetwork interface 1610 includes a multiport broadcast transmitter 1640and a receiver 1642. The multiport broadcast transmitter 1640 allows theprocessing system 1600 to broadcast multiple encoded stereoscopic datastreams each supporting different bit rates to various customer devices.In some embodiments the processing system 1600 transmits differentportions of a scene, e.g., 180 degree front portion, left rear portion,right rear portion etc., to customer devices via the multiport broadcasttransmitter 1640. Furthermore, via the multiport broadcast transmitter1640 the processing system 1600 also broadcasts composite depth map 1626to the one or more customer devices. While the multiport broadcasttransmitter 1640 is used in the network interface 1610 in someembodiments, still in some other embodiments the processing systemtransmits, e.g., unicasts or multicasts, the composite depth map and/orstereoscopic imaging content to individual customer devices.

The memory 1612 includes control routines 1614, image encoder(s) 1616, acomposite depth map generation module 1618, streaming controller 1620,received images 1621 of environment of interest, received depth maps ofthe environment of interest 1622, received stereoscopic image data 1624,generated composite depth map 1626 and encoded stereoscopic image data1628.

In some embodiments the modules are, implemented as software modules. Inother embodiments the modules are implemented in hardware, e.g., asindividual circuits with each module being implemented as a circuit forperforming the function to which the module corresponds. In still otherembodiments the modules are implemented using a combination of softwareand hardware.

The control routines 1614 include device control routines andcommunications routines to control the operation of the processingsystem 1600. The encoder(s) 1616 may, and in some embodiments do,include a plurality of encoders configured to encode received imagecontent, stereoscopic images of a scene and/or one or more sceneportions in accordance with the features of the invention. In someembodiments encoder(s) include multiple encoders with each encoder beingconfigured to encode a stereoscopic scene and/or partitioned sceneportions to support a given bit rate stream. Thus in some embodimentseach scene portion can be encoded using multiple encoders to supportmultiple different bit rate streams for each scene. An output of theencoder(s) 1616 is the encoded stereoscopic image data 1628 stored inthe memory for streaming to customer devices, e.g., playback devices.The encoded content can be streamed to one or multiple different devicesvia the network interface 1610.

The composite depth map generation module 1618 is configured to generatea composite depth map of the environment of interest from the depth mapsof the environment of interest 1622 received from the camera device1500. The generated composite depth map of the environment of interest1626 is an output of the composite depth map generation module 1618. Thestreaming controller 1620 is configured to control streaming of encodedcontent for delivering the encoded image content (e.g., at least aportion of encoded stereoscopic image data 1628) to one or more customerplayback devices, e.g., over the communications network 105. In variousembodiments the streaming controller 1620 is further configured tocommunicate, e.g., transmit, the composite depth map 1626 to one or morecustomer playback devices, e.g., via the network interface 1610.

The image generation module 1623 is configured to generate a first imagefrom at least one image captured by the light field camera, e.g.,received images 1621, the generated first image including a portion ofthe environment of interest which is not included in at least some ofthe images (e.g., stereoscopic image content 1624) captured by thestereoscopic cameras. In some embodiments the streaming controller 1620is further configured to transmit at least a portion of the generatedfirst image to one or more customer playback devices, e.g., via thenetwork interface 1610.

Received stereoscopic image data 1624 includes stereoscopic contentreceived from one or more stereoscopic cameras 103. Encoded stereoscopicimage data 1628 includes a plurality of sets of stereoscopic image datawhich have been encoded by the encoder(s) 1616 to support multipledifferent bit rate streams.

FIG. 15A illustrates a perspective view of an exemplary tower mountedsingle stereo camera pair rig 1700 in accordance with an exemplaryembodiment. The exemplary rig 1700 includes a base 1702 on which asupport tower 1704 is mounted. Over the support tower 1704 the camerarig support structure/mounting plate 1706 is placed. The supportstructure/mounting plate 1706 supports the stereoscopic camera pair 1710including a left eye camera 1711 and a right eye camera 1713 forcapturing left and right eye images respectively. The left eye camera1711 includes and/or is coupled to lens assembly 1712 and the right eyecamera 1713 includes and/or is coupled to lens assembly 1714. The camerarig 1700 also includes an interface 1708 via which the camera rig can becoupled to one or more external devices, e.g., for communicatingcaptures images and/or for remotely controlling the camera rig 1700.

FIG. 15B is a drawing 1725 illustrating a front view of the exemplarytower mounted single stereo camera pair rig 1700, with variousstructures and/or elements of the camera rig 1700 shown from a frontalperspective. The total height from a surface where the base 1702 reststo the top of the camera pair 1710 used in some embodiments is shown tobe 795.11 mm (millimeter).

FIG. 15C is a drawing 1750 illustrating a side view of the exemplarytower mounted single stereo camera pair rig 1700. The support tower 1704includes a fixed truss segment 1752 and a modular truss segment tosupport the camera pair 1710. FIG. 15C also shows various additionalconstructional features, dimensions and/or elevation adjustment valuesused in some embodiments.

FIG. 15D is a drawing 1760 illustrating a top view of the exemplarysingle stereo camera pair rig 1700. As can be seen in the top view shownin drawing 1760 the support base 1702 is a square shaped support havingthe dimensions 406.40 mm×406.40 in some embodiments.

FIG. 16A illustrates a top view of an exemplary triple stereo camerapair rig 1800 in accordance with some embodiments. As shown in drawing1800 the camera rig includes three pairs of stereoscopic camera pairsincluding camera pair 1802, 1806 and 1810 which are mounted on thesupport structure 1820. The electronic module assemblies 1830, 1834 and1838 which are coupled to their respective camera pairs are also shownin the figure.

FIG. 16B is a drawing 1825 illustrating a perspective view of theexemplary triple stereo camera pair rig 1800 with various featuresand/or elements of the camera rig 1800 being shown in more detail. Thefirst stereoscopic camera pair 1802 includes a left eye camera 1803 anda right eye camera 1804 and a sensor assembly 1820, the secondstereoscopic camera pair 1806 includes a left eye camera 1807 and aright eye camera 1808 and a sensor assembly 1822, and the thirdstereoscopic camera pair 1810 includes a left eye camera 1811 and aright eye camera 1812 and a sensor assembly 1824. The left and right eyecameras of each stereoscopic camera pair are coupled to theircorresponding lens assemblies. In some embodiments each of the sensorassemblies 1820, 1822, 1824 includes one sensor for each lens assembly,e.g., one for the left camera and one for the right camera. Theelectronic modules 1830, 1832 are coupled to the corresponding camerasensors in the sensor assembly 1820, the electronic modules 1834, 1836are coupled to the corresponding camera sensors in the sensor assembly1822 and the electronic modules 1838, 1840 are coupled to thecorresponding camera sensors in the sensor assembly 1824.

FIG. 16C is a drawing 1850 illustrating a side view of the exemplarytriple stereo camera pair rig 1800 with the camera pair 1806 being fullyvisible and only a single camera 1804 of the camera pair 1802 beingvisible from the side from which the camera rig 1800 is being shown.

FIG. 17 illustrates an exemplary tri stereoscopic camera rig 1900 inaccordance with an exemplary embodiment. The exemplary stereo camera rig1900 includes two stereoscopic camera pairs 1902, 1904 facing indifferent horizontal directions and a single camera 1906 facing upwardsin the vertical direction, e.g., for capturing sky images. The exemplarystereoscopic camera rig 1900 further includes a camera support platform1920 over which the cameras are mounted. Each of the stereoscopic camerapairs 1902, 1904 includes a left eye camera and a right eye camera forcapturing left and right eye images respectively. The left and right eyecameras of the stereoscopic camera pairs 1902, 1904 are coupled to theirrespective lens assemblies 1910, 1912 and 1914, 1916. The other camera1906 facing in the vertical direction is also coupled to its own lensassembly 1918.

FIG. 18 includes two different views of an exemplary tri stereoscopiccamera rig 2000 that includes a top upward facing camera and a bottomdownward facing camera in accordance with an exemplary embodiment. Thefirst drawing 2025 shows a top view of the exemplary tri stereoscopiccamera rig 2000 where the configuration of the camera rig 200 is shownfrom a top perspective. The exemplary camera rig 2000 includes threestereoscopic camera pairs 2002, 2004 and 2006, a top upward facingcamera 2008 and a bottom downward facing camera which is not visible indrawing 2025 but can be seen in drawing 2050 showing the side view. Inthe FIG. 18 embodiment equi-distant sensors, e.g., optical sensors ofthe illustrated cameras, are used throughout the camera array. Such aconfiguration provides for a highly uniform capture of depth andstereoscopic video from the full 360 horizontal view. In the illustratedconfiguration the stereo-correspondence is uniform for each of thesensors of the sensor pairs (camera pairs) and relative to the othersensors in the constellation.

FIG. 19 illustrates an exemplary two level stereo camera rig 2100 thatincludes six pairs of stereoscopic cameras arranged three camera pairsper level. The top level includes three stereoscopic camera pairs 2110,2112 and 2114 while the second bottom level includes another set ofthree stereoscopic camera pairs 2102, 2104 and 2106.

FIG. 20A illustrates a view of an exemplary quad stereoscopic camerapair rig 2200 in accordance with an exemplary embodiment. The exemplaryquad stereoscopic camera pair rig 2200 includes four stereo camera pairs2202, 2206, 2210 and 2214 arranged along four sides with the four sidesforming a rectangle, e.g., square, as shown. Each stereoscopic camerapair includes a left eye camera and a right eye camera for capturingleft and right eye images respectively. The stereoscopic camera pair2202 includes a left eye camera 2203 and a right eye camera 2204,stereoscopic camera pair 2206 includes a left eye camera 2207 and aright eye camera 2208, stereoscopic camera pair 2210 includes a left eyecamera 2211 and a right eye camera 2212 and the stereoscopic camera pair2214 includes a left eye camera 2213 and a right eye camera 2215.

FIG. 20B is a drawing 2225 illustrating a top view of the exemplary quadstereoscopic camera pair rig 2200 with more constructions featuresand/or dimensions shown for further detail. The left eye camera 2203 ofthe stereoscopic camera pair 2202 is coupled to lens assembly 2230 whilethe right eye camera 2204 of the stereoscopic camera pair 2202 iscoupled to lens assembly 2232. The left eye camera 2204 of thestereoscopic camera pair 2206 is coupled to lens assembly 2234 while theright eye camera 2208 of the stereoscopic camera pair 2206 is coupled tolens assembly 2236, the left eye camera 2211 of the stereoscopic camerapair 2210 is coupled to lens assembly 2238 while the right eye camera2212 is coupled to lens assembly 2240 and the left eye camera 2213 ofthe stereoscopic camera pair 2214 is coupled to lens assembly 2242 whilethe right eye camera 2215 is coupled to lens assembly 2244. As can beseen the directions in which each of the camera pairs face are differentand apart by 90 degrees in some embodiments.

FIG. 21A illustrates a view of an exemplary penta (five) stereoscopiccamera pair rig 2300 in accordance with an exemplary embodiment. Theexemplary five stereoscopic camera pair rig 2300 includes five stereocamera pairs 2302, 2304, 2306, 2308, and 2310 arranged along five sideswith the five sides forming a pentagon as shown. Each stereoscopiccamera pair includes a left eye camera and a right eye camera forcapturing left and right eye images respectively.

FIG. 21B is a drawing 2325 illustrating a top view of the exemplarypenta stereoscopic camera pair rig 2300 with more constructions featuresand/or dimensions shown for further detail in drawing 2325. The left eyecamera 2312 and the right eye camera 2314 of the stereoscopic camerapair 2302 are coupled to their respective lens assemblies 2313 and 2315,the left eye camera 2316 and the right eye camera 2318 of thestereoscopic camera pair 2304 are coupled to their respective lensassemblies 2317 and 2319, the left eye camera 2320 and the right eyecamera 2322 of the stereoscopic camera pair 2306 are coupled to theirrespective lens assemblies 2321 and 2323, the left eye camera 2324 andthe right eye camera 2326 of the stereoscopic camera pair 2308 arecoupled to their respective lens assemblies 2325 and 2327, and the lefteye camera 2328 and the right eye camera 2330 of the stereoscopic camerapair 2310 are coupled to their respective lens assemblies 2329 and 2331.As can be seen in FIG. 21B the directions in which each of the camerapairs face are different and apart by 72 degrees in some embodiments.

FIG. 22 is a drawing 2400 illustrating a side by side comparison of theexemplary penta stereoscopic camera pair rig 2300 on the left and theexemplary quad stereo camera pair rig 2200 shown on the right. Drawing2400 allows for better appreciation of the two camera pair rigs 2200 and2300 and the differences in their designs and configurations.

FIG. 23 illustrates a view of an exemplary hexa (six) stereoscopiccamera pair rig 2500 in accordance with an exemplary embodiment. Theexemplary six stereoscopic camera pair rig 2500 includes six stereocamera pairs 2502, 2504, 2506, 2508, 2510 and 2512 arranged along sixsides with the six sides forming a hexagon as shown. Each stereoscopiccamera pair includes a left eye camera and a right eye camera forcapturing left and right eye images respectively. The left eye camera2501 and the right eye camera 2503 of the stereoscopic camera pair 2502are coupled to their respective lens assemblies 2505 and 2507, the lefteye camera 2509 and the right eye camera 2511 of the stereoscopic camerapair 2504 are coupled to their respective lens assemblies 2513 and 2515,the left eye camera 2517 and the right eye camera 2519 of thestereoscopic camera pair 2506 are coupled to their respective lensassemblies which are not visible in FIG. 23, the left eye camera 2520and the right eye camera 2522 of the stereoscopic camera pair 2508 arecoupled to their respective lens assemblies 2523 and 2524, the left eyecamera 2526 and the right eye camera 2528 of the stereoscopic camerapair 2510 are coupled to their respective lens assemblies 2527 and 2529,the left eye camera 2530 and the right eye camera 2532 of thestereoscopic camera pair 2512 are coupled to their respective lensassemblies 2531 and 2533.

FIG. 24 is a drawing 2600 illustrating a top view of the hexa (six)stereoscopic camera pair rig 2500 with more constructions featuresand/or dimensions shown for further detail in drawing 2600. As shown thesix stereo camera pairs 2502, 2504, 2506, 2508, 2510 and 2312 arearranged along six sides of a hexagon. As can be seen in FIG. 24 thedirections in which each of the camera pairs face are different andapart by 60 degrees in some embodiments.

FIG. 25 illustrates a top view of an exemplary bi level ninestereoscopic camera pair rig 2700 in accordance with an exemplaryembodiment. The exemplary bi level nine stereoscopic camera pair rig2700 includes six pairs of stereoscopic cameras arranged on a top leveland another three pairs of stereoscopic cameras arranged on a lowerlevel. The top level includes six pairs of stereoscopic cameras 2702,2704, 2706, 2708, 2710 and 2712 arranged along six sides of a hexagon inthe similar manner as illustrated in FIGS. 23-24. The lower levelincludes three stereoscopic camera pairs 2714, 2716 and 2718 arranged inthe same or similar manner as in the case of camera rig 100 of FIGS.1-2. Each stereoscopic camera pair on the top level and the lower levelincludes a left eye camera and a right eye camera for capturing left andright eye images respectively. In some embodiments the directions inwhich each of the camera pairs 2702, 2704, 2706, 2708, 2710 and 2712face are different and apart by 60 degrees. In some embodiments thedirections in which each of the camera pairs 2714, 2716 and 2718 faceare different and apart by 120 degrees. In some embodiments the camerapairs 2702 and 2714 face in the same direction. In some embodiments thecamera pairs 2706 and 2716 face in the same direction. In someembodiments the camera pairs 2710 and 2718 face in the same direction.In some embodiments the sensor planes 2752, 2754, 2756, 2758, 2760 and2762 are within a distance of +/−1 mm from each other.

FIG. 26 illustrates an exemplary support structure 2800 including atripod comprising legs 2804, 2806, 2808 and a support ring or plate 2802to which a camera rig can be secured, e.g., by screws extending upwardthrough the screw holes in the support ring 2802. The support structuremay and sometimes does include feet 2814, 2816, 2808, one foot per legof the support structure 2800. which can be used to support variousexemplary camera rigs including, for example, the three sided camera rigshown in FIG. 3. In some embodiments the legs are welded or bolted tothe support ring to secure them in a fixed manner to the support ring.While the legs maybe secured in a fixed manner to the support ring 2802in some embodiments the legs 2804, 2806 and 2808 are attached to thesupport ring 2802 by hinges and/or pivots. Thus, in the hinged case thelegs maybe folded inward for transport and spread for use.

FIG. 27 illustrates an exemplary four legged support structure 2900which can be used to support one or more of the exemplary camera rigsshown in other figures including, for example, the four sided camerarigs shown in FIGS. 8 and 9. The four legged support structure 2900includes a support ring 2902 with holes through which screws may bepassed to secure a camera rig to the support structure 2900. The supportstructure 2900 includes four legs 2904, 2906, 2908 and 2910. The legsmaybe and sometime are secured to the support ring 2902 in a fixedmanner, e.g., by being welded or bolted to the support ring. In otherembodiments the legs 2904, 2906, 2908 and 2910 are secured to thesupport ring by a hinge or pivot allowing them to be moved inward fortransport.

FIG. 28 shows an image 2850 which shows the orientation of an exemplarystereoscopic camera rig 30 including 3 pairs of cameras relative with apreferred camera pair orientation relative to the support legs 2804,3806, 2808 of the tripod support structure 2800. In this configurationthe camera legs are intentionally positioned to be located to correspondto the midpoint between camera pairs of the rig 30. In this way, thelegs will appear in the field of view of the cameras, if at all, in theperipheral area where image capture is likely to be blurry or of lowerquality due to the use of fish eye lenses on the cameras of the camerapairs include din the rig 30. Thus, the notice-ability of the legs isminimized. In some embodiments the legs are of a known, predeterminedcolor. Image portions of the legs can and sometimes are identified basedon their color. location in a captured image and/or shape. In at leastsome embodiments prior to encoding and transmission, as part of imageper-pressing, image portions corresponding to the legs of the supportstructure are replaced with pixel which are the same as neighboringpixels of a capture image and/or which are generated based onneighboring pixels in a captured image. In this way to the extent that acaptured image includes a portion of a leg, the portion of the image,e.g., pixels, corresponding to the leg will be removed prior to encodingand transmission to an end playback device. The positioning of the legsso that they will appear, if at all, at an area of low image qualitydecreases the chance that the image processing used to remove the legswill occur at an area where the image processing to hide the legs islikely to be less noticeable than, for example, if the legs werepositioned directly in line with the forward orientation of a camera orcamera pair.

FIG. 29 shows an exemplary camera rig 1101 including 4 pairs of camerasused for stereoscopic image capture and a preferred orientation of thecamera rig to the support legs 2904, 2906, 2908, 2910 of the four leggedsupport structure 2900 shown in FIG. 29 that is used in some but not allembodiments. As with the three legged embodiment, the legs arepositioned so that the are offset to the left or right from the forwardlooking position of a camera or camera pair making the legs and/or theremoval of the legs from a captured image less noticeable to an endviewer viewing images in a 3D simulated environment, that uses imagescaptured by the camera rig as textures applied to a 3D model of theenvironment.

FIG. 30 is an illustration 3050 illustrating how the camera rig 1101 andsupport structure shown in FIG. 29 may appear during use with the camerarig secured to the support structure. The rig 1101 includes both anupward facing camera and a downward facing camera. A stereoscopic camerapair may be used, and in some embodiments is used, in place of theupward facing camera and downward facing camera. Using a camera rig andstand such as the one shown in FIG. 30, images of the ground, sky and360 degree environment can be captured at the same time. The images canbe and sometimes are, used as textures in a 3D simulated environment.

While steps are shown in an exemplary order it should be appreciatedthat in many cases the order of the steps may be altered withoutadversely affecting operation. Accordingly, unless the exemplary orderof steps is required for proper operation, the order of steps is to beconsidered exemplary and not limiting.

While various embodiments have been discussed, it should be appreciatedthat not necessarily all embodiments include the same features and someof the described features are not necessary but can be desirable in someembodiments.

While various ranges and exemplary values are described the ranges andvalues are exemplary. In some embodiments the ranges of values are 20%larger than the ranges discussed above. In other embodiments the rangesare 20% smaller than the exemplary ranges discussed above. Similarly,particular values may be, and sometimes are, up to 20% larger than thevalues specified above while in other embodiments the values are up to20% smaller than the values specified above. In still other embodimentsother values are used.

Some embodiments are directed a non-transitory computer readable mediumembodying a set of software instructions, e.g., computer executableinstructions, for controlling a computer or other device to encode andcompresses stereoscopic video. Other embodiments are embodiments aredirected a computer readable medium embodying a set of softwareinstructions, e.g., computer executable instructions, for controlling acomputer or other device to decode and decompresses video on the playerend. While encoding and compression are mentioned as possible separateoperations, it should be appreciated that encoding may be used toperform compression and thus encoding may, in some include compression.Similarly, decoding may involve decompression.

The techniques of various embodiments may be implemented using software,hardware and/or a combination of software and hardware. Variousembodiments are directed to apparatus, e.g., a image data capture andprocessing system. Various embodiments are also directed to methods,e.g., a method of image capture and/or processing image data. Variousembodiments are also directed to a non-transitory machine, e.g.,computer, readable medium, e.g., ROM, RAM, CDs, hard discs, etc., whichinclude machine readable instructions for controlling a machine toimplement one or more steps of a method.

Various features of the present invention are implemented using modules.Such modules may, and in some embodiments are, implemented as softwaremodules. In other embodiments the modules are implemented in hardware.In still other embodiments the modules are implemented using acombination of software and hardware. In some embodiments the modulesare implemented as individual circuits with each module beingimplemented as a circuit for performing the function to which the modulecorresponds. A wide variety of embodiments are contemplated includingsome embodiments where different modules are implemented differently,e.g., some in hardware, some in software, and some using a combinationof hardware and software. It should also be noted that routines and/orsubroutines, or some of the steps performed by such routines, may beimplemented in dedicated hardware as opposed to software executed on ageneral purpose processor. Such embodiments remain within the scope ofthe present invention. Many of the above described methods or methodsteps can be implemented using machine executable instructions, such assoftware, included in a machine readable medium such as a memory device,e.g., RAM, floppy disk, etc. to control a machine, e.g., general purposecomputer with or without additional hardware, to implement all orportions of the above described methods. Accordingly, among otherthings, the present invention is directed to a machine-readable mediumincluding machine executable instructions for causing a machine, e.g.,processor and associated hardware, to perform one or more of the stepsof the above-described method(s).

Numerous additional variations on the methods and apparatus of thevarious embodiments described above will be apparent to those skilled inthe art in view of the above description. Such variations are to beconsidered within the scope.

What is claimed:
 1. An image capture system, comprising: a first lightfield camera facing in a first direction; and a first stereoscopiccamera pair including at least a first camera and a second camera forcapturing left and right views, respectively.
 2. The system of claim 1,wherein said first light field camera includes a fish eye lens; whereinsaid first camera includes a fisheye lens; and wherein said secondcamera includes a fish eye lens.
 3. The system of claim 1, furthercomprising: a second light field camera facing in a second direction;and a second stereoscopic camera pair including at least a third cameraand a fourth camera.
 4. The system of claim 3, wherein said second lightfield camera includes a fish eye lens; wherein said third cameraincludes a fisheye lens; and wherein said fourth camera includes a fisheye lens.
 5. The system of claim 3, further comprising: a third lightfield camera facing in a third direction; and a third stereoscopiccamera pair including at least a fifth camera and a sixth camera.
 6. Thesystem of claim 5, further comprising: a fourth light field camerafacing in a fourth direction; and a fourth stereoscopic camera pairincluding at least a seventh camera and an eighth camera.
 7. The systemof claim 6, further comprising: a ninth camera facing in an upwardvertical direction; and wherein the first, second third and fourthdirections are generally horizontal directions.
 8. The system of claim7, further comprising: a tenth camera facing in a downward verticaldirection opposite the direction in which the ninth camera is pointed;and wherein the first, second, third, fourth, sixth, seventh, ninth andtenth cameras are equidistant from one another.
 9. The system of claim1, further comprising: a second light field camera facing in a seconddirection; and a third light field camera facing in a third direction,said first, second and third directions being different.
 10. The systemof claim 9, further comprising: a fourth light field camera facing in afourth direction.
 11. The system of claim 6, wherein said first, second,third and fourth light field cameras each include a processor forgenerating a depth map from an image captured by the light field camerain which the processor is included.
 12. The system of claim 11, furthercomprising: a processor configured to receive the depth mapscorresponding to portions of the environment and combine the depth mapsto generate an environmental depth map.
 13. The system of claim 11further comprising: memory for storing images captured by said first,second and third, light field cameras, said environmental depth map andimages captured from said first, second and third stereoscopic camerapairs, at least some of the images captured by the light field camerahaving been captured while a stereoscopic camera was capturing some ofsaid stored images captured by the stereoscopic camera pairs.
 14. Animage capture and processing method, comprising: capturing images usinga first light field camera facing in a first direction; and capturingleft and right eye images using a first stereoscopic camera pairincluding at least a first camera and a second camera for capturing leftand right views, respectively.
 15. The method of claim 14, furthercomprising: capturing images using a second light field camera facing ina second direction; capturing left and right eye images using a secondstereoscopic camera pair including at least a third camera and a fourthcamera for capturing left and right views, respectively.
 16. The methodof claim 15, further comprising: capturing images using a third lightfield camera facing in a third direction; and capturing left and righteye images using a third stereoscopic camera pair including at least afifth camera and a sixth camera.
 17. The method of claim 16, furthercomprising: generating an environmental depth map from images capturedby said first, second, and third light field cameras.
 18. The method ofclaim 16, further comprising: capturing images using a fourth lightfield camera facing in a fourth direction; capturing images using afourth stereoscopic camera pair including at least a seventh camera andan eighth camera; and generating an environmental depth map from imagescaptured by said first, second, third and fourth light field cameras.19. The method of claim 16, further comprising: generating stereoscopicimage data from a light field image captured by one of said first,second and third light field cameras, said stereoscopic image dataincluding image data corresponding to a portion of the environment whichwas not captured by a camera of a stereoscopic camera pair.